Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (#786) #788

jowg-amazon · 2023-12-19T01:01:26Z

Description

An ecs field may have multiple raw fields associated with it. Currently, in the getMappingsView api, the ecs field will only map to one raw field. The new implementation will automatically map the ecs field to whatever field is present in the index. If multiple raw fields are present in an index, it will automatically map to one of the raw fields and add the other(s) into the unmappedIndexFields list.

This PR also removes duplicates from the unmappedFieldAliases list.

Issues Resolved

#786

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

amsiglan · 2023-12-19T08:11:39Z

src/main/java/org/opensearch/securityanalytics/mapper/MapperService.java

+                                        aliasMappingFields.put(mapping.getEcs(), Map.of("type", "alias", "path", mapping.getRawField()));
+                                    }
+                                } else {
+                                    aliasMappingFields.put(mapping.getEcs(), Map.of("type", "alias", "path", mapping.getRawField()));


This is same as the if case. Not sure how it would work differently.

The first if clause checks if the alias mapping fields already contains the ecs key, if it's true then it checks to make sure that the raw field path that we want to insert is consistent with the one in pathsOfApplyableAliases. If it's not consistent, then we skip and do not override the mapping. If the key doesn't exist in the map in the first place, then we directly insert the mapping.

codecov · 2023-12-19T19:06:05Z

Codecov Report

Attention: Patch coverage is 0% with 16 lines in your changes are missing coverage. Please review.

Project coverage is 24.80%. Comparing base (8ef0a3f) to head (1a2c07b).

Files	Patch %	Lines
...search/securityanalytics/mapper/MapperService.java	0.00%	16 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #788      +/-   ##
============================================
- Coverage     24.82%   24.80%   -0.02%     
  Complexity     1030     1030              
============================================
  Files           277      277              
  Lines         12717    12727      +10     
  Branches       1401     1404       +3     
============================================
  Hits           3157     3157              
- Misses         9296     9306      +10     
  Partials        264      264

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

engechas · 2024-02-22T02:34:16Z

src/main/java/org/opensearch/securityanalytics/mapper/MapperService.java

                        Map<String, Map<String, String>> aliasMappingFields = new HashMap<>();
                        XContentBuilder aliasMappingsObj = XContentFactory.jsonBuilder().startObject();
                        for (LogType.Mapping mapping : requiredFields) {
                            if (allFieldsFromIndex.contains(mapping.getOcsf())) {
                                aliasMappingFields.put(mapping.getEcs(), Map.of("type", "alias", "path", mapping.getOcsf()));
                            } else if (mapping.getEcs() != null) {
-                                aliasMappingFields.put(mapping.getEcs(), Map.of("type", "alias", "path", mapping.getRawField()));
+                                // check if aliasMappingFields already contains a key
+                                if (aliasMappingFields.containsKey(mapping.getEcs())) {


Minor: we're already very nested here, we could create a private method to handle this logic rather than nesting further

refactored out the method for better readability

eirsep · 2024-02-27T22:24:51Z

src/main/java/org/opensearch/securityanalytics/mapper/MapperService.java

+
+                        // filter out aliases that were included in applyableAliases already
+                        List<String> filteredUnmappedFieldAliases = setOfUnmappedFieldAliases.stream()
+                                .filter(e -> applyableAliases.contains(e) == false)


nit alway write this condition as false == <boolean> for quicker readability and less confusion

Signed-off-by: Joanne Wang <jowg@amazon.com>

opensearch-trigger-bot · 2024-03-08T01:39:15Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-788-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 689760e897294530dea0d1181c2539e9b607d23c
# Push it to GitHub
git push --set-upstream origin backport/backport-788-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-788-to-2.x.

… mapping view API (opensearch-project#786) (opensearch-project#788) * field mapping changes Signed-off-by: Joanne Wang <jowg@amazon.com> * add integ test Signed-off-by: Joanne Wang <jowg@amazon.com> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <jowg@amazon.com> * add comments Signed-off-by: Joanne Wang <jowg@amazon.com> * fix integ tests Signed-off-by: Joanne Wang <jowg@amazon.com> * moved logic to method for better readability Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com>

… mapping view API (#786) (#788) (#898) * field mapping changes * add integ test * turn unmappedfieldaliases as set and add integ test * add comments * fix integ tests * moved logic to method for better readability --------- Signed-off-by: Joanne Wang <jowg@amazon.com>

Signed-off-by: Joanne Wang <jowg@amazon.com> (cherry picked from commit 4d4f5e3) Co-authored-by: Joanne Wang <jowg@amazon.com> Reduce log level for informative message (opensearch-project#203) (opensearch-project#833) Signed-off-by: Enrico Tröger <enrico.troeger@uvena.de> Co-authored-by: Enrico Tröger <enrico.troeger@uvena.de> Updated alert creation following common-utils PR 584. (opensearch-project#837) (opensearch-project#839) Signed-off-by: AWSHurneyt <hurneyt@amazon.com> (cherry picked from commit 8adb9c3) Co-authored-by: AWSHurneyt <hurneyt@amazon.com> Release notes for 2.12.0 (opensearch-project#834) (opensearch-project#841) * release notes for 2.12 Signed-off-by: Joanne Wang <jowg@amazon.com> * update release notes Signed-off-by: Joanne Wang <jowg@amazon.com> * update release notes Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com> (cherry picked from commit 414484a) Co-authored-by: Joanne Wang <jowg@amazon.com> Remove blocking calls and change threat intel feed flow to event driven (opensearch-project#871) (opensearch-project#876) * remove actionGet() and change threat intel feed flow to event driven Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix javadocs Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * revert try catch removals Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * use action listener wrap() in detector threat intel code paths Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add try catch Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> (cherry picked from commit 172d58d) Co-authored-by: Surya Sashank Nistala <snistala@amazon.com> Fail the flow the when detectot type is missing in the log types index (opensearch-project#845) (opensearch-project#857) Signed-off-by: Megha Goyal <goyamegh@amazon.com> (cherry picked from commit 8d19912) Co-authored-by: Megha Goyal <56077967+goyamegh@users.noreply.github.com> [BUG] ArrayIndexOutOfBoundsException for inconsistent detector index behavior (opensearch-project#843) (opensearch-project#858) * Catch ArrayIndexOutOfBoundsException when detector is missing Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Add a check on SearchHits.getHits() length Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Remove index out of bounds exception Signed-off-by: Megha Goyal <goyamegh@amazon.com> --------- Signed-off-by: Megha Goyal <goyamegh@amazon.com> (cherry picked from commit 0ef8543) Co-authored-by: Megha Goyal <56077967+goyamegh@users.noreply.github.com> Backport opensearch-project#873 and opensearch-project#789 (opensearch-project#895) * support object fields in aggregation based sigma rules (opensearch-project#789) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Pass rule field names in doc level queries during monitor/creation. Remove blocking actionGet() calls (opensearch-project#873) * pass query field names in doc level queries during monitor creation/updation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * remove actionGet() and change get index mapping call to event driven flow Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix chained findings monitor Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add finding mappings Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * remove test messages from logs Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * revert build.gradle change Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (opensearch-project#786) (opensearch-project#788) (opensearch-project#898) * field mapping changes * add integ test * turn unmappedfieldaliases as set and add integ test * add comments * fix integ tests * moved logic to method for better readability --------- Signed-off-by: Joanne Wang <jowg@amazon.com> Add throw for empty strings in rules with modifier contains, startwith, and endswith (opensearch-project#860) (opensearch-project#896) * add validation for empty strings with contains, startswith and endswith modifiers * throw exception if empty string with contains, startswith, or endswith * change var name * add modifiers to log --------- Signed-off-by: Joanne Wang <jowg@amazon.com> Add an "exists" check for "not" condition in sigma rules (opensearch-project#852) (opensearch-project#897) * test design Signed-off-by: Joanne Wang <jowg@amazon.com> * working version Signed-off-by: Joanne Wang <jowg@amazon.com> * cleaning up Signed-off-by: Joanne Wang <jowg@amazon.com> * testing Signed-off-by: Joanne Wang <jowg@amazon.com> * working version Signed-off-by: Joanne Wang <jowg@amazon.com> * working version Signed-off-by: Joanne Wang <jowg@amazon.com> * refactored querybackend Signed-off-by: Joanne Wang <jowg@amazon.com> * working on tests Signed-off-by: Joanne Wang <jowg@amazon.com> * fixed alerting and finding tests Signed-off-by: Joanne Wang <jowg@amazon.com> * fix correlation tests Signed-off-by: Joanne Wang <jowg@amazon.com> * working all tests Signed-off-by: Joanne Wang <jowg@amazon.com> * moved test and changed alias for adldap Signed-off-by: Joanne Wang <jowg@amazon.com> * added more tests Signed-off-by: Joanne Wang <jowg@amazon.com> * cleanup code Signed-off-by: Joanne Wang <jowg@amazon.com> * remove exists flag Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com> (cherry picked from commit 656a5fe) Co-authored-by: Joanne Wang <jowg@amazon.com> Add goyamegh as a maintainer (opensearch-project#868) (opensearch-project#899) Signed-off-by: Megha Goyal <goyamegh@amazon.com> Refactor invocation of Action listeners in correlations (opensearch-project#880) (opensearch-project#900) * Refactor invocation of Action listeners in correlations * Close hanging tasks in correlations workflow * Logging finding id and monitor id in error logs --------- Signed-off-by: Megha Goyal <goyamegh@amazon.com> Add search request timeouts for correlations workflows (opensearch-project#893) (opensearch-project#901) * Reinstating more leaks plugged-in for correlations workflows Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Add search timeouts to all correlation searches Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Fix logging and exception messages Signed-off-by: Megha Goyal <goyamegh@amazon.com> * Change search timeout to 30 seconds Signed-off-by: Megha Goyal <goyamegh@amazon.com> --------- Signed-off-by: Megha Goyal <goyamegh@amazon.com> (cherry picked from commit 75c4429) Co-authored-by: Megha Goyal <56077967+goyamegh@users.noreply.github.com>

opensearch-trigger-bot · 2024-03-11T23:36:24Z

The backport to 2.9 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.9 2.9
# Navigate to the new working tree
cd .worktrees/backport-2.9
# Create a new branch
git switch --create backport/backport-788-to-2.9
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 689760e897294530dea0d1181c2539e9b607d23c
# Push it to GitHub
git push --set-upstream origin backport/backport-788-to-2.9
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.9

Then, create a pull request where the base branch is 2.9 and the compare/head branch is backport/backport-788-to-2.9.

opensearch-trigger-bot · 2024-03-11T23:36:24Z

The backport to 2.5 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.5 2.5
# Navigate to the new working tree
cd .worktrees/backport-2.5
# Create a new branch
git switch --create backport/backport-788-to-2.5
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 689760e897294530dea0d1181c2539e9b607d23c
# Push it to GitHub
git push --set-upstream origin backport/backport-788-to-2.5
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.5

Then, create a pull request where the base branch is 2.5 and the compare/head branch is backport/backport-788-to-2.5.

opensearch-trigger-bot · 2024-03-11T23:36:24Z

The backport to 2.11 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.11 2.11
# Navigate to the new working tree
cd .worktrees/backport-2.11
# Create a new branch
git switch --create backport/backport-788-to-2.11
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 689760e897294530dea0d1181c2539e9b607d23c
# Push it to GitHub
git push --set-upstream origin backport/backport-788-to-2.11
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.11

Then, create a pull request where the base branch is 2.11 and the compare/head branch is backport/backport-788-to-2.11.

opensearch-trigger-bot · 2024-03-11T23:36:25Z

The backport to 2.7 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.7 2.7
# Navigate to the new working tree
cd .worktrees/backport-2.7
# Create a new branch
git switch --create backport/backport-788-to-2.7
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 689760e897294530dea0d1181c2539e9b607d23c
# Push it to GitHub
git push --set-upstream origin backport/backport-788-to-2.7
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.7

Then, create a pull request where the base branch is 2.7 and the compare/head branch is backport/backport-788-to-2.7.

… mapping view API (opensearch-project#786) (opensearch-project#788) * field mapping changes Signed-off-by: Joanne Wang <jowg@amazon.com> * add integ test Signed-off-by: Joanne Wang <jowg@amazon.com> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <jowg@amazon.com> * add comments Signed-off-by: Joanne Wang <jowg@amazon.com> * fix integ tests Signed-off-by: Joanne Wang <jowg@amazon.com> * moved logic to method for better readability Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com>

* support object fields in aggregation based sigma rules (#789) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (#786) (#788) * field mapping changes Signed-off-by: Joanne Wang <jowg@amazon.com> * add integ test Signed-off-by: Joanne Wang <jowg@amazon.com> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <jowg@amazon.com> * add comments Signed-off-by: Joanne Wang <jowg@amazon.com> * fix integ tests Signed-off-by: Joanne Wang <jowg@amazon.com> * moved logic to method for better readability Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com> * Fix get mappings view API incorrectly returning ecs path (#867) * add logic and integ tests to not add duplicate to log-types-config index Signed-off-by: Joanne Wang <jowg@amazon.com> * change naming for existingFieldMapping and change contains to equals Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com> * fix integ test (#918) Signed-off-by: Joanne Wang <jowg@amazon.com> * get all findings as part of findings API enhancement (#803) * get all findings as part of findings API enhancement Signed-off-by: Riya Saxena <riysaxen@amazon.com> * findingsAPI feature enhancements (address comments to prev PR) Signed-off-by: Riya Saxena <riysaxen@amazon.com> * findingsAPI feature enhancements (address comments to prev PR) Signed-off-by: Riya Saxena <riysaxen@amazon.com> * added support for param in Finding API Signed-off-by: Riya Saxena <riysaxen@amazon.com> * added detectionType as param for Findings API enhancements Signed-off-by: Riya Saxena <riysaxen@amazon.com> * added few tests to validate findings by params Signed-off-by: Riya Saxena <riysaxen@amazon.com> * added test for searchString param in FindingsAPI Signed-off-by: Riya Saxena <riysaxen@amazon.com> * adding addiional params findingIds, startTime and endTime as findings API enhancement Signed-off-by: Riya Saxena <riysaxen@amazon.com> * added params in getFindingsByDetectorId func * changed the startTime and endTime req input format * fix merge conflixt * fix integ test failures in findings API * fix integ tests * fix integ tests for findings Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> --------- Signed-off-by: Riya Saxena <riysaxen@amazon.com> Signed-off-by: Riya <69919272+riysaxen-amzn@users.noreply.github.com> Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> * Feature findings api enhancements (#914) * get all findings as part of findings API enhancement Signed-off-by: Riya Saxena <riysaxen@amazon.com> * findingsAPI feature enhancements (address comments to prev PR) Signed-off-by: Riya Saxena <riysaxen@amazon.com> * findingsAPI feature enhancements (address comments to prev PR) Signed-off-by: Riya Saxena <riysaxen@amazon.com> * added support for param in Finding API Signed-off-by: Riya Saxena <riysaxen@amazon.com> * added detectionType as param for Findings API enhancements Signed-off-by: Riya Saxena <riysaxen@amazon.com> * added few tests to validate findings by params Signed-off-by: Riya Saxena <riysaxen@amazon.com> * added test for searchString param in FindingsAPI Signed-off-by: Riya Saxena <riysaxen@amazon.com> * adding addiional params findingIds, startTime and endTime as findings API enhancement Signed-off-by: Riya Saxena <riysaxen@amazon.com> * added params in getFindingsByDetectorId func * changed the startTime and endTime req input format * fix merge conflixt * fix integ test failures in findings API * fix integ tests * refactored the logic Signed-off-by: Riya Saxena <riysaxen@amazon.com> * remove unused imports * address the pr comments Signed-off-by: Riya Saxena <riysaxen@amazon.com> * address pr comments Signed-off-by: Riya Saxena <riysaxen@amazon.com> * SA integ tests fix * SA integ tests fix * fix integ tests for findings Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * fix conflixt errors Signed-off-by: Riya Saxena <riysaxen@amazon.com> * fix conflixt errors Signed-off-by: Riya Saxena <riysaxen@amazon.com> * fix conflixt errors Signed-off-by: Riya Saxena <riysaxen@amazon.com> * fix conflixt errors Signed-off-by: Riya Saxena <riysaxen@amazon.com> * fix integ tests Signed-off-by: Riya Saxena <riysaxen@amazon.com> * fix integ tests Signed-off-by: Riya Saxena <riysaxen@amazon.com> * fix integ tests Signed-off-by: Riya Saxena <riysaxen@amazon.com> * fix flaky integ tests Signed-off-by: Riya Saxena <riysaxen@amazon.com> * address pr comments Signed-off-by: Riya Saxena <riysaxen@amazon.com> --------- Signed-off-by: Riya Saxena <riysaxen@amazon.com> Signed-off-by: Riya <69919272+riysaxen-amzn@users.noreply.github.com> Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> * fix findings api integ tests Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: Joanne Wang <jowg@amazon.com> Signed-off-by: Riya Saxena <riysaxen@amazon.com> Signed-off-by: Riya <69919272+riysaxen-amzn@users.noreply.github.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Co-authored-by: Riya <69919272+riysaxen-amzn@users.noreply.github.com>

* Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (#786) (#788) * field mapping changes Signed-off-by: Joanne Wang <jowg@amazon.com> * add integ test Signed-off-by: Joanne Wang <jowg@amazon.com> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <jowg@amazon.com> * add comments Signed-off-by: Joanne Wang <jowg@amazon.com> * fix integ tests Signed-off-by: Joanne Wang <jowg@amazon.com> * moved logic to method for better readability Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com> * support object fields in aggregation based sigma rules (#789) Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> * Pass rule field names in doc level queries during monitor/creation. Remove blocking actionGet() calls (#873) * pass query field names in doc level queries during monitor creation/updation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * remove actionGet() and change get index mapping call to event driven flow Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix chained findings monitor Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add finding mappings Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * remove test messages from logs Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * revert build.gradle change Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * version bump Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * AggregationBackendTests fix Signed-off-by: Riya Saxena <riysaxen@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com> Signed-off-by: Subhobrata Dey <sbcd90@gmail.com> Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Riya Saxena <riysaxen@amazon.com> Co-authored-by: Joanne Wang <jowg@amazon.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> Co-authored-by: Surya Sashank Nistala <snistala@amazon.com>

… mapping view API (#786) (#788) * field mapping changes Signed-off-by: Joanne Wang <jowg@amazon.com> * add integ test Signed-off-by: Joanne Wang <jowg@amazon.com> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <jowg@amazon.com> * add comments Signed-off-by: Joanne Wang <jowg@amazon.com> * fix integ tests Signed-off-by: Joanne Wang <jowg@amazon.com> * moved logic to method for better readability Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com>

* Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (#786) (#788) * field mapping changes Signed-off-by: Joanne Wang <jowg@amazon.com> * add integ test Signed-off-by: Joanne Wang <jowg@amazon.com> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <jowg@amazon.com> * add comments Signed-off-by: Joanne Wang <jowg@amazon.com> * fix integ tests Signed-off-by: Joanne Wang <jowg@amazon.com> * moved logic to method for better readability Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com> * Fix get mappings view API incorrectly returning ecs path (#867) * add logic and integ tests to not add duplicate to log-types-config index Signed-off-by: Joanne Wang <jowg@amazon.com> * change naming for existingFieldMapping and change contains to equals Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com> * fix integ test (#918) Signed-off-by: Joanne Wang <jowg@amazon.com> * fix detector writeTo() method missing fields (#695) * fix detector writeTo() method missing fields Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix test Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * removing threatIntel related code Signed-off-by: Riya Saxena <riysaxen@amazon.com> * removing threatIntel related code Signed-off-by: Riya Saxena <riysaxen@amazon.com> * ignore flaky tests Signed-off-by: Riya Saxena <riysaxen@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com> Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Signed-off-by: Riya Saxena <riysaxen@amazon.com> Co-authored-by: Joanne Wang <jowg@amazon.com> Co-authored-by: Surya Sashank Nistala <snistala@amazon.com>

… mapping view API (#786) (#788) (#890) * field mapping changes Signed-off-by: Joanne Wang <jowg@amazon.com> * add integ test Signed-off-by: Joanne Wang <jowg@amazon.com> * turn unmappedfieldaliases as set and add integ test Signed-off-by: Joanne Wang <jowg@amazon.com> * add comments Signed-off-by: Joanne Wang <jowg@amazon.com> * fix integ tests Signed-off-by: Joanne Wang <jowg@amazon.com> * moved logic to method for better readability Signed-off-by: Joanne Wang <jowg@amazon.com> --------- Signed-off-by: Joanne Wang <jowg@amazon.com> (cherry picked from commit 689760e) Co-authored-by: Joanne Wang <jowg@amazon.com>

jowg-amazon requested review from amsiglan, AWSHurneyt, getsaurabh02, lezzago, praveensameneni, sbcd90 and eirsep as code owners December 19, 2023 01:01

amsiglan reviewed Dec 19, 2023

View reviewed changes

jowg-amazon force-pushed the field_bug786 branch from 04d8812 to c18c79a Compare February 20, 2024 17:45

jowg-amazon requested a review from engechas as a code owner February 20, 2024 17:45

engechas previously approved these changes Feb 22, 2024

View reviewed changes

eirsep reviewed Feb 27, 2024

View reviewed changes

eirsep previously approved these changes Feb 27, 2024

View reviewed changes

jowg-amazon added 6 commits February 27, 2024 15:14

field mapping changes

54fc914

Signed-off-by: Joanne Wang <jowg@amazon.com>

add integ test

585478b

Signed-off-by: Joanne Wang <jowg@amazon.com>

turn unmappedfieldaliases as set and add integ test

a556667

Signed-off-by: Joanne Wang <jowg@amazon.com>

add comments

87f4912

Signed-off-by: Joanne Wang <jowg@amazon.com>

fix integ tests

b7da1c8

Signed-off-by: Joanne Wang <jowg@amazon.com>

moved logic to method for better readability

1a2c07b

Signed-off-by: Joanne Wang <jowg@amazon.com>

jowg-amazon dismissed stale reviews from eirsep and engechas via 1a2c07b February 27, 2024 23:50

jowg-amazon force-pushed the field_bug786 branch from c18c79a to 1a2c07b Compare February 27, 2024 23:50

jowg-amazon requested review from goyamegh and riysaxen-amzn as code owners February 27, 2024 23:50

riysaxen-amzn approved these changes Feb 27, 2024

View reviewed changes

engechas approved these changes Mar 7, 2024

View reviewed changes

jowg-amazon merged commit 689760e into opensearch-project:main Mar 7, 2024
12 of 18 checks passed

jowg-amazon added backport 2.x backport 2.12 labels Mar 7, 2024

opensearch-trigger-bot bot mentioned this pull request Mar 7, 2024

[Backport 2.12] Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (#786) #890

Merged

jowg-amazon added backport 2.x and removed backport 2.x labels Mar 8, 2024

jowg-amazon mentioned this pull request Mar 8, 2024

[Backport 2.x] Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (#786) #889 #898

Merged

5 tasks

jowg-amazon added backport 2.5 backport 2.7 backport 2.9 backports to 2.9 backport 2.11 labels Mar 11, 2024

jowg-amazon mentioned this pull request Mar 14, 2024

[Backport 2.11] Backport #789, #788, #867, and #918 to 2.11 #925

Closed

5 tasks

jowg-amazon mentioned this pull request Mar 14, 2024

[Backport 2.11] Backport #788 to 2.11 #929

Closed

5 tasks

jowg-amazon mentioned this pull request Mar 15, 2024

[Backport 2.11] Backport #789, #788, #867, and #918 to 2.11 #931

Merged

5 tasks

riysaxen-amzn mentioned this pull request Mar 18, 2024

Backport 2.9 fix #937

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (#786) #788

Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (#786) #788

jowg-amazon commented Dec 19, 2023

amsiglan Dec 19, 2023

jowg-amazon Dec 19, 2023

codecov bot commented Dec 19, 2023 •

edited

Loading

engechas Feb 22, 2024

jowg-amazon Feb 27, 2024

eirsep Feb 27, 2024

jowg-amazon Feb 27, 2024

opensearch-trigger-bot bot commented Mar 8, 2024

opensearch-trigger-bot bot commented Mar 11, 2024

opensearch-trigger-bot bot commented Mar 11, 2024

opensearch-trigger-bot bot commented Mar 11, 2024

opensearch-trigger-bot bot commented Mar 11, 2024

Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (#786) #788

Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (#786) #788

Conversation

jowg-amazon commented Dec 19, 2023

Description

Issues Resolved

Check List

amsiglan Dec 19, 2023

Choose a reason for hiding this comment

jowg-amazon Dec 19, 2023

Choose a reason for hiding this comment

codecov bot commented Dec 19, 2023 • edited Loading

Codecov Report

engechas Feb 22, 2024

Choose a reason for hiding this comment

jowg-amazon Feb 27, 2024

Choose a reason for hiding this comment

eirsep Feb 27, 2024

Choose a reason for hiding this comment

jowg-amazon Feb 27, 2024

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Mar 8, 2024

opensearch-trigger-bot bot commented Mar 11, 2024

opensearch-trigger-bot bot commented Mar 11, 2024

opensearch-trigger-bot bot commented Mar 11, 2024

opensearch-trigger-bot bot commented Mar 11, 2024

codecov bot commented Dec 19, 2023 •

edited

Loading